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^uMpary 

A  study  is  made  of  u  class  of  laathenLitical  problems  connected  with  physical 
situations  which  re'-.uire  that  a  finite  or  unbo\indad  secinence  of  operations  l>e 
performed  for  the  purpose  of  achieving  a  desired  result.  The  only  case  considered 
her«  is  that  in  which  each  operation  performs  a  mapping  of  the  parametric  space 
onto  itself. 


§1,  Introduction. 

W*  are  interested  in  a  class  of  mathematical  problems  connected  with  physical 


situations  which  require  that  a  bounded  or  unbounded  sequence  of  operations  be 
perforaed  for  the  purpose  of  achieving  a  desired  result.  The  result  of  each 
individual  operation  is  a  stochastic  event  which  yields  information  to  be  used 
in  planning  the  subsequent  operations.  We  consider  here  only  the  case  where 
each  operation,  perforaed  upon  a  system  whose  state  is  specified  by  a  set  of 
parameters,  'has  the  effect  of  converting  one  state  into  another.  This  last  is 
equivalent  to  saying  that  each  operation  performs  a  mapping  of  the  parameter 
space  onto  itself. 

The  two  principal  problems  encountered  in  situations  of  this  type  are  those 
of  maximizing  the  yield  obtained  in  a  given  time,  or  of  minimizing  the  tiae 
required  to  accomplish  a  certain  task.  Since  we  are  dealing  with  a  sequence  of 
stochastic  events  we  shall  have  to  e^loy  the  metrics  of  probability  theory, 
and  the  two  types  of  criteria  we  will  use  are: 

1.  The  expected  yield,  or  time. 

2.  The  probability  of  achieving  a  certain  goal,  i.e.,  the  probability 
that  the  yield  is  greater  than  a  given  quantity,  or  that  the  time 
is  less  than  a  given  time. 

We  shall  see  below  that  the  choice  of  an  appropriate  criterion  plays  a 

A 

decisive  irole  in  determining  the  form  of  the  solution  to  the  problems  we  consider. 

An  extensive  utilization  in  the  theory  of  statistical  inference  of  the 
fundamental  idea  that  the  results  of  the  preceding  operations  should  be  used  to 
guide  the  course  of  the  subsequent  operations  is  due  to  Wald  in  his  theory  of 
sequential  analysis. 


HowtT«r,  Um  fVill  ranf«  of  problaas  of  this  tjp*  does  not  seen  to  have  been 
appreciated,  nor  does  the  class  of  matheaatleal  problem  encountered  seen  to  have 
been  diecussed  at  mny  length.  The  purpose  of  the  present  paper  is  to  indicate 
some  of  the  vast  number  of  probleias  which  maj  be  treated  from  the  above  view¬ 
point  and  to  indicate  briefly  the  nature  of  the  solutions.  A  more  complete 
exposition  will  be  given  elsewhere. 

Some  of  the  probleias  we  discuss  here  are  related  to  those  discussed  by 
Arrow,  Blackwell  and  Girshick,^^^  and  Wald  and  Wolfowits,^^^. 

All  of  these  problems  could  be  included  as  particular  cases  of  an  abstract 
formulation.  However,  it  is  important  to  postpone  this  until  a  large  number  of 
individual  problems  have  been  solved,  since  certain  indigenous  features  of  each 
problem  will  facilitate  its  solution. 

What  ties  the  variety  of  problems  we  consider  together  is  a  point  of  view 
which  affords  an  immediate  foothold.  In  order  to  preeent  this  concept  we  must 
first  discuss  what  is  meant  by  a  solution  of  a  problem  of  this  type.  By  a 
solution  we  mean  a  set  of  rules  which  tell  us  which  operation  to  perform  at  every 
stage  in  every  situation.  Since  the  nmber  of  possibilities  in  some  of  the 
problems  involving  unbounded  sequences  is  quite  large,  a  listing  of  all  possible 
optimal  sequences  is  neither  feasible  nor  elegant  nor  useful. 

To  avoid  these  unattractive  features  we  begin  by  observing  that  we  have 
stipulated  that  the  result  of  each  operation  is  to  change  the  system  into  a 
similar  system  determined  by  different  parameters.  In  light  of  this,  the 
question  arises  as  to  whether  or  not  it  night  not  be  sufficient  to  prescribe 
merely  the  best  first  move  in  each  situation  that  arises.  We  are  immediately, 
howeveri  confronted  by  a  difficulty.  Our  original  purpose  was  to  optimize 
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according  to  a  certain  criterion,  of  the  type  given  above.  It  may  not  be  true 
that  after  the  first  stage,  the  optimal  continuation  is  to  be  made  according  to 
the  same  criterion. 

We  are  led  naturally  to  direct  our  attention  to  those  criteria  which  possess 
a  certain  invariant  property,  namely,  after  any  finite  number  of  operations,  the 
optimal  continuation  is  obtained  by  employing  the  original  criterion  to  guide  the 
subsequent  operations. 

Some  examples  of  criteria  having  this  property  are; 

1.  Maxini;;e  the  expected  yield,  or  minimise  the  expe/^ted  time. 

2,  Maximize  the  probability  of  success. 

Instances  of  criteria  not  possessing  tWs  property  are: 

1.  Maximize  the  expected  yield  at  the  end  of  M  stages. 

2.  Maximize  the  probability  of  success  in  N  or  fswer  stages. 

Let  us  now  turn  to  the  discussion  of  sods  specific  problems. 

^2.  Determination  of  the  state  of  a  physical  system. 

Frequently  it  is  necessary  to  determine  by  testing  the  parameters  of 
state  of  a  physical  system  whei*e  it  is  not  possible  to  examine  the  entire  parameter 
space  at  one  time.  Since  each  test  requires  time,  the  question  arises  naturally 
of  arranging  the  testing  program  to  consume  the  least  time.  In  this  case,  we 
may  use  either  the  criterion  of  minimizing  expected  time,  or  the  probability  of 
determining  the  state  in  time  less  than  T.  It  is  clear  that  we  would  test 
for  the  most  likely  state  first.  Wt  make  the  problem  more  interesting  if  we 
attach  to  each  state  a  probability  that  the  testing  device  will  not  register,  or 
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what  is  more  drastic,  that  it  will  be  destroyed  before  registering  ai\y  information. 

We  may  also  consider  the  situation  where  each  test  disturbs  the  system. 

These  problems  are  more  difficult  than  those  above. 

As  a  simple  model  of  a  problem  of  the  first  type,  consider  the  problem  of 
finding  a  ball  which  is  known  to  be  in  one  of  N  boxes,  with  a  probability  distribu¬ 
tion  Pj^  that  it  is  in  the  box.  Let  be  the  probability,  i.xlependent  of 

previous  trials,  that  we  are  unable  to  look  into  the  k  box  when  we  axamine  it  and 
assume  for  simplicity  that  each  examination  consumes  one  unit  of  tine.  We  wish  to 
determine  the  testing  procedure  which  minimizes  the  expected  time  required  to  find 

*.  \ 


We  may  merely  wish  to  locate  the  box  containing  it  or  we  may  wish  actually  to 
obtain  or  view  the  ball  itself.  It  is  clear  that  for  a  sizable  nmber  of  boxes 
there  will  be  little  difference  between  the  expected  times  required  to  accomplish 
either  purpose. 

If  we  use  the  criterion  of  expected  time  to  obtain  or  v^ew  the  ball,  tne 
solution  is  extraordinarily  simple:  at  each  stage,  axamine  the  box  for  which 
Pjj(l  -  is  largest.  Here  (Pj^)  is  the  probability  distribution  before  the 
test. 


Let  us  sketch  the  method  of  proof.  Let 
time  required  using  the  optimal  procedure, 
yields  the  ftuictional  equation 


E||(Pi,  Pj*  ••*,  Pij)  be  the  expected 
Then  an  enweiation  of  possibilities 


(1)  Vl"  ^2- 


p^)  •  min 


<I,[l  •  Pj, 


♦  (1  -  q^) 


1 


where  the  term  coi-reepoivling  to  p^  is  omitted  in  Having  established  by 
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direct  calculation  the  result  for  N  •  2,  it  is  easy  to  proceed  inductively  and 
obtain  the  result  stated  above. 

If  we  use  the  alternative  criterion,  that  of  Bsrely  locating  the  ball, 
certainly  a  nore  natural  one,  we  obtain  the  same  Ainctional  equation.  However, 
the  case  N  ■  2  is  now  quite  different  due  to  the  fact  that  having  looked  in 
either  box  and  not  having  found  it  inmediately  locates  the  ball  in  the  other  box. 
Hence  for  N  ■  2  one  would  examine  the  box  for  which  is  largest,  unlsss  ■  0 
or  1.  There  is  an  interesting  discontinuity  here  which  persists  throughout.  The 
optiaal  procedure  for  p^  /  0,  1  does  not  furnish  as  — ^0  or  1  the  optimal 
procedures  for  these  cases.  We  will  find  in  the  case  of  N  boxes  that  the 
(Pl#  P2*  ***«  P)j)  sp^ee  is  divided  into  N  regions,  1^,  i  ■  1,  ***,  N,  having  ths 
property  that  if  P2*  ***>  ^  the  la  to  be  ammmined  first. 

These  regions  maj  be  computed  readily,  beginning  with  N  *  3>  using  the 
functional  equation  (1). 

This  functional  equation  seems  to  be  of  a  type  not  previously  encountered. 

In  view  of  the  above  result,  we  see  that  the  choice  of  an  appropriate 
criterion  is  of  great  importance.  It  is  conceivable  that  in  other  problems  of 
sequential  analysis  a  slight  change  in  the  criterion  lugr  materially  simplify  the 
mathematical  treatment . 

Results  similar  to  the  above  hold  if  we  introduce  probabilities  of  breakage 
of  the  instrument  and  consider  as  a  criterion  the  probability  of  finding  or 
obtaining  the  ball. 

^3.  Testing  for  information. 

Suppose  that  we  own  one  testlr^  device  and  have  a  number  of  objects  to  be 
tested,  each  of  which  possesses  a  certain  amount  of  information.  We  may  assume 
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that  there  is  no  possibility  of  breakage  of  the  instrument  and  ask  for  the  testing 
procedure  which  yields  the  maximum  infonaation  in  a  (^iven  time,  or  we  may  assme 
that  the  instrument  is  breakable  and  ask  for  the  testing  policy  which  yields  the 
maximum  information  before  the  instrunent  is  destroyed.  In  the  first  case  assume 
further  that  there  is  a  set  of  probability  distributions  which  govern  the 
information  obtained  from  testing  the  objsct.  It  can  be  shown  inductively 
via  the  functional  equation,  or  using  the  fact  that  the  order  of  testing  is 
immaterial  in  this  case  once  one  has  decided  to  test  the  k^^  object  a  fixed 
number  of  times,  that  an  optimal  procedure  is  one  which  miximizes  the  expected 
information  obtained  at  each  step,^^^. 


In  the  second  case  where  we  attach  to  eacli  object  a  probability  that  the 
instrument  is  destroyed  in  the  course  of  testing,  it  is  no  longer  true  that  one 
maximizes  the  expected  information  obtained  at  each  step.  Rather,  one  naximizes 
the  where  is  the  ejqjected  information  obtained  from  testing  the  k^^ 

object. 

Since  the  form  of  the  solution  is  independent  of  N  (which  is  not  obvious 
beforehand),  it  is  sufficient  to  determine  the  solution  for  N  •  2.  Here  an 
intuitive  method  rapidly  leads  to  the  solution.  The  only  parameters  which  change 
from  test  to  test  are  the  amounts  of  information  contained  in  each  object,  which 


we  call  I  ,  II  .  We  suspect  that  the  (v^,  v^)  positive  quadrant  will  be 
divided  into  two  regions,  R^,  with  the  property  that  if  (v^^,  v^)  t  the  i^^ 
object  is  tested  on  the  first  step.  These  regions  will  be  determined  if  we  can 


find  the  common  boundary,  which  is  the  locusof  points  ( v,  ,  v„)  for  which  it  is  a 
matter  of  indifference  as  to  which  object  is  tested  first.  If  we  write  down  the 
functional  equation  correspondir^  to  (1)  of  §2  and  equate  the  right-hand  sides. 


we  find  that  we  obtain  little  infomation  since  the  unknown  function  itself  appears. 
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One  step  further,  however,  yields  the  solution.  The  boundaiy  curve  C  separating 
and  should  have  the  property  that  if  (v^^,  V2)  €  C  and  the  first  object  is 
tested,  then  the  new  (v^,  V2)  point  is  in  R2*  and  conversely.  Hence  the 
symbolic  equation  (I)  •  (II)  leads  to  (I*  II)  ■  (II  •  I).  When  this  equation  is 
written  out  the  unknown  fwictions  on  both  sides  cancel  and  we  are  left  with  a 
linear  equation  in  v^^  eind  which  is  precisely 

The  problem  becomes  more  interesting  *and  decidedly  more  difficult  if  we 
allow  the  possibility  of  one  instnnent  testing  several  objects  in  one  operation 
or  consider  more  testing  devices  being  used  simultaneously.  A  consideration  of 
the  simplest  such  casu  shows  very  clearly  the  difficulties  that  are  encountered 
in  attempting  a  general  theory. 

Consider  two  objects,  I  and  II,  and  a  testinf  device  which  may  be  used  to 
test  either  one  separately  or  both  together.  V.'e  consider  thi*ee  possibilities 
only,  no  information,  complete  information,  or  a  fixed  percentage  of  the 
information  obtained.  Let 

(1)  ■  probability  instris^nt  is  not  destroyed  testing  i^^  object, 

and  if  not  destroyed, 

r^  *  probability  of  complete  information, 

Sj^  -  probability  of  no  information 

tj^  ■  probability  of  0  of  the  information  obtained. 

Let  f(v^,  V2)  "  expected  information  obtained  using  optimal  policy.  Then 

(2)  f(v, ,  v^)  -  max 

1,2 

where  [l — >2]  indicates  the  similar  expression  obtained  from  testing  II  first. 
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W«  would  •xpoctf  from  what  has  pracadad,  that  there  will  be  three  regions  of 
the  positive  quadrant  R^,  R21  the  properties  that  if  (v^,  v^)  R^,  ^2 

or  R^2»  tests  I,  II,  or  both.  If  we  attempt  to  use  the  previous  arguments 
to  datarmina  tha  boundary,  we  find  that  the  method  fails,  since  a  point  on  a 
boundary  can  move  into  either  of  two  regions  and  it  is  not  simple  to  determine 
which. 

To  overcome  this  difficulty,  we  use  a  continuity  method,  taking  0  as  a 
parameter  wMch  varies  between  0  and  1,  At  each  extreme  it  is  easy  to  determine 
the  three  regions.  As  o  varies  we  may  follow  the  boundaries  and  show  that  for 
all  o  these  three  regions  exist.  The  boundaries  will  depend  upon  the  proba¬ 
bilities  listed  in  (1). 

Notice  that  in  the  above  we  have  taken  the  o's  to  be  equal  in  all  three 
operations.  It  would  be  desirable  to  take  them  as  distinct.  We  can  use  one  0 
for  tests  of  I  and  II  and  another  for  simultaneous  tests  of  I  and  II  and  the 
method  applies  equally.  However,  if  all  the  o's  are  distinct,  not  only  loes  our 
method  fail,  but  there  are  examples  showing  that  the  method  falls,^^^. 

The  essential  fact  that  makes  this  problem  tractable  is  a  periodicity  in  the 
tests  that  occurs  in  the  special  case,  but  not  the  general  case  of  distinct  o's. 

§4.  Other  problems. 

Let  us  in  this  section  merely  mention  some  other  problems  of  this  general 

class. 

An  Investment  Problem; 

Suppose  a  person  is  left  a  sxua  of  money  to  live  on  for  the  rest  of  his 
life.  He  would  like  to  use  all  the  money  before  he  dies,  but  he  loes  not  wish 
to  be  left  penniless  while  alive.  How  should  he  apportion  his  money  between 
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conauaptlon  ard  investaent  if  he  wishes  to  consume  the  maximum  (expected)  amount 
of  money  before  he  dies? 

Problems  of  this  type  are  also  encountered  in  a  treatment  of  optimal 
gambling  methods,  or  in  the  theory  of  games  in  general  if  one  assumes  that  one 
at  least  of  the  players  has  a  finite  amount  of  money. 

A  Logistics  Problem; 


th 


Suppose  that  we  have  n  porta,  A^,  **•»  A^,  and  let  the  i  port  have 


.th 


a^  ships  available  at  the  initial  time,  and  a^^^  shiploads  to  be  sent  to  the  J 
port,  a^^^  ■  0,  How  does  one  route  the  ships  so  as  to  minimize  the  total  transit 
time? 


For  the  case  where  equal  times  are  consumed  sailing  between  any  two  ports, 
a  very  elegant  solution  has  been  given  by  E.  V,'.  Paxson,  depending  only  upon 
conservation  principles. 

Finally,  we  note  that  there  are  many  problems  in  organization  theory  relating 
to  the  communication  of  information  and  the  performance  of  many-person  tasks 
which  ’aay  be  considered  to  be  problems  of  the  general  type  considered  here* 


I ' 
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